Add support for away timeout #801

paulheinrichs-jb · 2025-11-04T17:59:53Z

Description

The framework didn't support the away state, so this PR adds support for this. I've done my best to document the logical paths of python ensuring to match verbatim here.

Changes Made

Condition/Trigger	Python Logic	JavaScript Implementation	Status	Notes
Timer Initialization
`user_away_timeout` is `None`	Return early, no timer set	✅ Check `null \|\| undefined`, return early	✅ COMPLETE	Exact match
`room_io.subscribed_fut.done()` is `False`	Skip timer (user hasn't joined)	✅ Check `!this.roomIO.isParticipantAvailable`	✅ COMPLETE	Matches Python logic
Valid timeout + participant available	Set timer with `loop.call_later(timeout, callback)`	✅ Use `setTimeout(callback, timeout * 1000)`	✅ COMPLETE	Platform-appropriate equivalent
User State Transitions
User → 'speaking'	Cancel timer via `_cancel_user_away_timer()`	✅ Cancel in `_updateUserState()`	✅ COMPLETE	Exact logic match
User → 'listening' + Agent 'listening'	Set timer via `_set_user_away_timer()`	✅ Set in `_updateUserState()`	✅ COMPLETE	Exact logic match
User → 'away'	Cancel timer	✅ Cancel in `_updateUserState()`	✅ COMPLETE	Exact logic match
Agent State Transitions
Agent → 'speaking'	Cancel timer	✅ Cancel in `_updateAgentState()`	✅ COMPLETE	Exact logic match
Agent → 'listening' + User 'listening'	Set timer	✅ Set in `_updateAgentState()`	✅ COMPLETE	Exact logic match
Agent → other states	Cancel timer	✅ Cancel in `_updateAgentState()`	✅ COMPLETE	Exact logic match
User Activity Detection
`_user_input_transcribed()` called with `is_final=True`	Reset from 'away' to 'listening'	✅ Reset in `_onUserInputTranscribed()` with `isFinal`	✅ COMPLETE	Exact logic match
`_user_input_transcribed()` called with `is_final=False`	No state change	✅ Only act on `isFinal`	✅ COMPLETE	Exact logic match
Session Lifecycle
`aclose()` called	Cancel timer in `_aclose_impl()`	✅ Cancel in `closeImpl()`	✅ COMPLETE	Proper cleanup
Timer Management
`_set_user_away_timer()` called multiple times	Cancel existing, set new	✅ `_cancelUserAwayTimer()` then set new	✅ COMPLETE	Prevents multiple timers
`_cancel_user_away_timer()` called when no timer	No-op	✅ Check `!== null` before clearing	✅ COMPLETE	Safe operation
Error Conditions
Timer callback execution	Call `_update_user_state('away')`	✅ Call `_updateUserState('away')`	✅ COMPLETE	Exact match
Edge Cases
Session closed while timer active	Timer cancelled in cleanup	✅ Timer cancelled in `closeImpl()`	✅ COMPLETE	Memory leak prevention
Multiple rapid state changes	Each change cancels/sets timer appropriately	✅ Each state change handles timer correctly	✅ COMPLETE	Race condition safe
User speaks immediately after going away	State reset to 'listening'	✅ State reset in `_onUserInputTranscribed()`	✅ COMPLETE	Immediate recovery

GMT20251104-174726_Clip_Paul.Heinrichs.s.Clip.11_04_2025.mp4

Pre-Review Checklist

Build passes: All builds (lint, typecheck, tests) pass locally
AI-generated code reviewed: Removed unnecessary comments and ensured code quality
Changes explained: All changes are properly documented and justified above
Scope appropriate: All changes relate to the PR title, or explanations provided for why they're included

Testing

Automated tests added/updated (if applicable)
All tests pass
Make sure both restaurant_agent.ts and realtime_agent.ts work properly (for major changes)

Additional Notes

Notably the following shows up in my demo, but is related to the inference STT

(node:13823) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. MaxListeners is 10. Use events.setMaxListeners() to increase limit
    at [kNewListener] (node:internal/event_target:566:17)
    at [kNewListener] (node:internal/abort_controller:240:24)
    at EventTarget.addEventListener (node:internal/event_target:679:23)
    at waitForAbort (.../agents-js/agents/src/utils.ts:828:10)
    at recv (.../agents-js/agents/src/inference/stt.ts:332:55)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Task.fn (.../agents-js/agents/src/inference/stt.ts:369:11)
    at async run (.../agents-js/agents/src/utils.ts:433:14)

Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.

changeset-bot · 2025-11-04T17:59:56Z

🦋 Changeset detected

Latest commit: 500a5f5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 14 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugins-test	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

toubatbrian · 2025-11-06T09:20:32Z

agents/src/voice/agent_session.ts

+  }
+
+  private _onUserInputTranscribed(ev: UserInputTranscribedEvent): void {
+    if (this.userState === 'away' && ev.isFinal) {


Can we discard checking ev.isFinal? So we can turn away state back much quicker as soon as user starts speaking

toubatbrian · 2025-11-06T09:20:41Z

agents/src/voice/agent_session.ts

+  private _onUserInputTranscribed(ev: UserInputTranscribedEvent): void {
+    if (this.userState === 'away' && ev.isFinal) {
+      this.logger.debug('User returned from away state due to speech input');
+      this._updateUserState('listening');


Should this be "speaking"?

Re this and the above comment, happy to make this change.
I was however, trying to keep this in parity of the python implementation

https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_session.py#L1109-L1113

Do you feel like we should still be adjusting?

oh I see, in this case let's keep this in parity with python.

examples/src/idle_user_timeout_example.ts

toubatbrian

Add a few comments, overall LGTM. cc @Shubhrakanti

Also, please create a changeset for this PR, thanks!

toubatbrian · 2025-11-07T19:12:26Z

examples/src/idle_user_timeout_example.ts

+    const userPresenceTask = async (controller: AbortController): Promise<void> => {
+      for (let i = 0; i < 3; i++) {
+        if (controller.signal.aborted) return;
+
+        const reply = await session.generateReply({
+          instructions: 'The user has been inactive. Politely check if the user is still present.',
+        });
+
+        await reply.waitForPlayout();
+
+        try {
+          await delay(10000, { signal: controller.signal });
+        } catch {
+          return;
+        }
+      }
+
+      if (!controller.signal.aborted) {
+        await session.close();
+      }
+    };
+
+    session.on(voice.AgentSessionEventTypes.UserStateChanged, (event) => {
+      logger.info({ event }, 'User state changed');
+
+      if (task) {
+        task.cancel();
+      }


this looks pretty good!

Add support for away timeout

906cfae

toubatbrian reviewed Nov 6, 2025

View reviewed changes

examples/src/idle_user_timeout_example.ts Outdated Show resolved Hide resolved

toubatbrian reviewed Nov 6, 2025

View reviewed changes

paulheinrichs-jb added 4 commits November 6, 2025 08:32

Leverage task

16164a6

Refactor idle timeout

71f8ff8

patch agents for away event

2b51ecd

One more utility function

500a5f5

paulheinrichs-jb requested a review from toubatbrian November 7, 2025 18:05

toubatbrian reviewed Nov 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for away timeout #801

Add support for away timeout #801

Uh oh!

paulheinrichs-jb commented Nov 4, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Nov 4, 2025 •

edited

Loading

Uh oh!

toubatbrian Nov 6, 2025

Uh oh!

toubatbrian Nov 6, 2025

Uh oh!

paulheinrichs-jb Nov 6, 2025

Uh oh!

toubatbrian Nov 7, 2025

Uh oh!

Uh oh!

toubatbrian left a comment •

edited

Loading

Uh oh!

toubatbrian Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add support for away timeout #801

Are you sure you want to change the base?

Add support for away timeout #801

Uh oh!

Conversation

paulheinrichs-jb commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes Made

Pre-Review Checklist

Testing

Additional Notes

Uh oh!

changeset-bot bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

toubatbrian Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

toubatbrian Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

paulheinrichs-jb Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

toubatbrian Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

toubatbrian left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toubatbrian Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

paulheinrichs-jb commented Nov 4, 2025 •

edited

Loading

changeset-bot bot commented Nov 4, 2025 •

edited

Loading

toubatbrian left a comment •

edited

Loading